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(57) Methods for communicating between an appli- 
cation and an ink divider object (which stores ink strokes 
to be divided into groups) may include: (a) issuing a di- 
vide request to the ink divider object, optionally by the 
application; (b) in response to the divide request, calling 
a divide method, which groups the stored ink strokes 
into one or more groupings of strokes having a first pre- 
determined granularity (e.g., words, lines, paragraphs, 
sentences, drawings, etc.); and (c) making information 
regarding the one or more groupings of strokes availa- 
ble to the application. This "information" made available 
to the application may include, for example, the actual 
groupings of the strokes, the number of stroke group- 
ings having the first predetermined granularity, machine 
generated text corresponding to the stroke groupings, 
or the like. The results of the divide method may be 
stored in an inkdivision result object. In some examples, 
the ink division result object may include the originally 
divided strokes and allow retrieval of groupings of 
strokes of various different granularities. This invention 
also relates to systems for performing the above meth- 
ods and various data structures used in performing 
these methods. 



_SYSTEMMEM0RY_ 
( ROM) 140 



OTHER 
PROGRAM 197 
MODULES 







N 


PROCESSING 
UNIT 


110 


VtOEO 
ADAPTER 



XL 



3T 



HARD 
DISK 
INTERFACE 



MAGNETIC 
DISK OR IV E 
INTERFACE 



OPTICAL 
DRIVE 
INTERFACE 



SERIAL 
PORT 
INTERFACE 



„lJ— S?0 I— !_k 191 



T t 

\&i» 19 r© 



\7~ 



LOCAL AREA 



APPLICATION .„ 
PROGRAMS 180 



wrQ tort 



/ J MEMORY ) 



Figure 1 



CL 
LD 



Printed by Jouve, 75001 PARIS (FR) 



1 



EP 1 450 294 A1 



2 



Description 
Related Applications 

[0001] In general, the present invention may be used 
in conjunction with the systems and methods disclosed 
in the following patent applications: 

(a) U.S. PatentAppln. No. 10/143,865, filed May 14, 
2002, entitled "Handwriting Layout Analysis of 
Freeform Digital Ink Input;" 

(b) U.S. PatentAppln. No. 10/143,864, filed May 14, 
2002, entitled "Classification Analysis of Freeform 
Digital Ink Input;" 

(c) U.S. Patent Appln. No. 1 0/1 43,804, filed May 1 4, 
2002, entitled "An Incremental System for Real 
Time Digital Ink Analysis;" and 

(d) U.S. Patent Appln. No. 10/184,108, filed June 
28, 2002, entitled "Interfacing With Ink." 

Each of these co-pending U.S. patent applications is en- 
tirely incorporated herein by reference. 

Field of the Invention 

[0002] Aspects of the present invention relate to sys- 
tems, methods, and computer-readable media that fa- 
cilitate communication between an application program 
and electronic ink, including various ink and ink divider 
objects. Some examples of such systems, methods, 
and computer-readable media enable application pro- 
gram or client code access to ink stroke groupings of 
various granularity to improve performance of the appli- 
cation programs and allow improved interaction of these 
programs and their associated code with digital ink. 

Background 

[0003] Typical computer systems, especially compu- 
ter systems using graphical user interface (GUI) sys- 
tems, such as Microsoft WINDOWS, are optimized for 
accepting user input from one or more discrete input de- 
vices, such as a keyboard for entering text and a point- 
ing device (e.g., a mouse with one or more buttons), for 
driving the user interface. The ubiquitous keyboard and 
mouse interface provides for fast creation and modifica- 
tion of documents, spreadsheets, database fields, draw- 
ings, photos and the like. However, in some respects, 
there is a significant gap in the flexibility provided by the 
keyboard and mouse interface as compared with the 
non-computer (i.e., conventional) pen and paper. With 
conventional pen and paper, a user may edit a docu- 
ment, write notes in a margin, and draw pictures and 
other shapes, and the like. In some instances, a user 
may prefer to use a pen to mark-up a document rather 



than review the document on a computer screen be- 
cause of the ability to freely make notes outside of the 
confines of the keyboard and mouse interface. 
[0004] Some computer systems permit users to draw 
5 on a screen. For example, the Microsoft READER ap- 
plication allows users to add electronic ink (also referred 
to herein as "ink" or "digital ink") to a document. The 
system stores the ink and provides it to a user when re- 
quested. Other applications (for example, drawing ap- 
10 plications as known in the art associated with the Palm 
3.x and 4.x and PocketPC operating systems) permit the 
capture and storage of drawings. Also, various drawing 
applications, such as Corel Draw, and photo and editing 
applications, such as Photoshop, may be used with sty- 
's lus based input products, such as the Wacom tablet 
product. These drawings include other properties asso- 
ciated with the ink strokes used to make up the draw- 
ings. For instance, line width and color may be stored 
with the ink. One goal of these systems is to replicate 
20 the look and feel of physical ink being applied to a piece 
of paper. 

[0005] While computer systems that accept electronic 
ink are known, at present time their availability and use- 
fulness, in at least some respects, are somewhat limit- 

25 ed. To further increase their availability and usefulness, 
application programs must include code that allows in- 
teraction and interfacing with the electronic ink. Accord- 
ingly, an application programming interface ("API") that 
allows code writers to readily, flexibly, and consistently 

30 interact and interface with various different groupings of 
ink would be very useful to those who wish to write code 
for application programs that interact in some manner 
with electronic ink. 



[0006] Applications that implement freeform drawing 
surfaces where, for example, users can input and inter- 
act with electronic ink on a page, are faced with the chal- 
lenge of determining at what scope to store and manip- 
ulate the strokes that the user provides. The straightfor- 
ward approaches for an application developer are: (1) 
treat each stroke individually or (2) treat all strokes on 
a page, or in a given editing session, together. Each of 
these approaches, however, has serious practical limi- 
tations in terms of ease of use for the end user as well 
as compatibility with existing document layout code. The 
ideal approach for an application, but one that is ordi- 
narily quite difficult to implement, is to treat the strokes 
in groups comprising words, lines, or paragraphs. This 
approach has great benefits for ease of use, compati- 
bility, making possible improved handwriting recognition 
and many other features, etc. This invention produces 
APIs that application developers can use to easily get 
these benefits without having to determine themselves 
how to group the strokes, thus removing a major diffi- 
culty of this approach. 

[0007] Aspects of the present invention relate to sys- 
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terns and methods for making information available to 
an application program. These systems and methods 
may include: storing a plurality of ink strokes; issuing or 
receiving a divide request; in response to the divide re- 
quest, grouping the stored ink strokes into one or more 
groupings of strokes having at least a first predeter- 
mined granularity; and making information regarding the 
one or more groupings of strokes available to the appli- 
cation program. The "information" made available to the 
application program may include, for example, at least 
one of the one or more groupings of strokes; information 
indicating a number of groupings of strokes having the 
first predetermined granularity; and machine-generated 
text that corresponds to at least one of the one or more 
groupings of strokes. The strokes may be grouped into 
various different granularities, such as groups contain- 
ing words, lines, paragraphs, sentences, drawings, etc. 
The grouping action also may group the strokes into 
groupings of more than one different granularity, and it 
may be repeated after the ink stroke set is changed, for 
example, by adding, deleting, moving, resizing, or oth- 
erwise modifying one or more strokes. Application pro- 
gram code also can provide various types of parsing in- 
formation to the parser during operation of the methods 
described above, such as setting the recognizer to use 
during parsing, setting a language to be used during 
parsing, setting a desired granularity into which the 
strokes will be parsed, setting expected line heights for 
lines of text included in the ink strokes, and the like. 
[0008] Additional aspects of the present invention re- 
late to systems and methods for communicating be- 
tween an application and an inkdivider object that stores 
ink strokes to be divided into groups. In some examples, 
the systems and methods include: (a) issuing a divide 
request to the ink divider object, optionally by the appli- 
cation; (b) in response to the divide request, calling a 
divide method, which groups the stored ink strokes into 
one or more groupings of strokes having at least a first 
predetermined granularity (e.g., words, lines, para- 
graphs, sentences, drawings, etc.); and (c) making in- 
formation regarding the one or more groupings of 
strokes available to the application. The results of the 
divide method may be stored in an ink division result 
object. In some examples, the ink division result object 
may include (and allow application program access to) 
the originally divided ink strokes and may allow retrieval 
of groupings of strokes of various different granularities. 
In additional examples of the invention, the divide meth- 
od may use a predetermined or preset language char- 
acteristic associated with the ink strokes to assist in bet- 
ter defining the groupings of ink strokes. 
[0009] Still additional aspects of the present invention 
relate to computer-readable media having computer-ex- 
ecutable instructions stored thereon for performing the 
various methods generally described above. Additional 
aspects of the present invention relate to computer- 
readable media having data structures stored thereon 
for various inkdivider objects, ink division result objects, 



ink division units objects, and ink division unit objects. 
[0010] These and other features and aspects of the 
present invention will be more apparent upon consider- 
ation of the following detailed description and the draw- 
5 ings. 

Brief Description of the Drawings 

[0011] The foregoing Summary, as well as the follow- 

10 ing Detailed Description, is better understood when read 
in conjunction with the accompanying drawings, which 
are included by way of example, and not by way of lim- 
itation with regard to the claimed invention. 
[0012] Fig. 1 shows aschematic diagram of ageneral- 

15 purpose digital computing environment that can be used 
to implement various aspects of the invention. 
[0013] Fig. 2 shows a plan view of a stylus-based 
computing system that can be used in accordance with 
various aspects of the present invention. 

20 [0014] Fig. 3 shows a general overview of an example 
of a parsing system and/or method that may be used in 
conjunction with examples of this invention. 
[0015] Fig. 4 shows adiagram generally explaining in- 
cremental parsing processing that may be used in con- 

25 junction with examples of this invention. 

[0016] Fig. 5 illustrates an example of various layout 
analysis steps that may be used in conjunction with ex- 
amples of this invention. 

[0017] Figs. 6A and 6B illustrate examples of parse 
30 tree data structures that may be obtained, for example, 
using a layout analysis engine that performs the steps 
illustrated in Fig. 5. 

[0018] Fig. 7 illustrates components and features of 
an InkDivider object used in some examples of the 
35 present invention. 

[0019] Fig. 8 illustrates components and features of 
an InkDivisionResult object used in some examples of 
the present invention. 

[0020] Fig. 9 illustrates components and features of 
40 an InkDivisionUnits object used in some examples of the 
present invention. 

[0021] Fig. 10 illustrates components and features of 
an InkDivisionUnit object used in some examples of the 
present invention. 
45 [0022] Fig. 11 illustrates another example of an Ink- 
Divider object used in some examples of the present in- 
vention. 

Detailed Description 

50 

[0023] The following description is divided into sub- 
sections to assist the reader. The sub-sections include: 
Terms; General-Purpose Computer; General Back- 
ground on Ink Layout Analysis and Classification Anal- 
55 ysis; The Ink Divider Object and API; Operation of the 
Ink Divider Object and API; Application Programming In- 
terfaces; An Alternative Ink Divider Object; and Conclu- 
sion. 
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I. TERMS 

[0024] I nk - A sequence or set of one or more strokes, 
optionally with properties. A sequence of strokes may 
include strokes in an ordered form. The sequence may 
be ordered by the time captured or by where the strokes 
appear on a page. Other orders also are possible. A set 
of strokes may include sequences of strokes or unor- 
dered strokes or any combination thereof. Ink may be 
expanded to include additional properties, methods, 
trigger events, and the like. 

[0025] Ink object - A data structure storing one or 
more ink strokes, with or without properties, methods, 
and/or events. 

[0026] Stroke - A sequence or set of captured points. 
For example, when rendered, a sequence of points may 
be connected with lines. Alternatively, a stroke may be 
represented as a point and a vector in the direction of 
the next point. In short, a stroke is intended to encom- 
pass any representation of points or segments relating 
to ink, irrespective of the underlying representation of 
points and/or what connects the points. 
[0027] Point - Information defining a location in space. 
For example, points may be defined relative to a cap- 
turing space (for example, points on a digitizer), a virtual 
ink space (the coordinates in a space into which cap- 
tured ink is represented or stored), and/or display space 
(the points or pixels of a display device). 
[0028] Render - The process of determining how 
graphics and/or ink are to be displayed, whether on a 
screen, printed, or output into another data format. 
[0029] Inking Session - A period of time from when an 
application begins creating or editing ink until a parser 
(e.g., an ink divider object) is called upon to examine 
the inkstrokes and return parsed ink entities. The parser 
may be called multiple times during a given inking ses- 
sion, and strokes may be added, deleted, or otherwise 
modified in between calls of the parser. 

II. GENERAL-PURPOSE COMPUTER 

[0030] Fig. 1 illustrates a schematic diagram of an ex- 
ample of a conventional general-purpose digital com- 
puting environment that may be used to implement var- 
ious aspects of the present invention. In Fig. 1 , a com- 
puter 100 includes a processing unit or system 110, a 
system memory 1 20, and a system bus 1 30 that couples 
various system components including the system mem- 
ory to the processing unit 1 1 0. The system bus 1 30 may 
be any of several types of bus structures including a 
memory bus or memory controller, a peripheral bus, and 
a local bus using any of a variety of bus architectures. 
The system memory 120 includes read only memory 
(ROM) 140 and random access memory (RAM) 150. 
[0031] A basic input/output system 160 (BIOS), con- 
taining the basic routines that help to transfer informa- 
tion between elements within the computer 100, such 
as during start-up, is stored in the ROM 140. The com- 



puter 1 00 also includes a hard disk drive 1 70 for reading 
from and writing to a hard disk (not shown), a magnetic 
disk drive 1 80 for reading from or writing to a removable 
magnetic disk 190, and an optical disk drive 191 for 
5 reading from or writing to a removable optical disk 1 92, 
such as a CD ROM or other optical media. The hard disk 
drive 170, magnetic disk drive 180, and optical diskdrive 
1 91 are connected to the system bus 1 30 by a hard disk 
drive interface 192, a magnetic disk drive interface 193, 
and an optical disk drive interface 1 94, respectively. The 
drives and their associated computer-readable media 
provide nonvolatile storage of computer readable in- 
structions, data structures, program modules, and other 
data for the personal computer 1 00. It will be appreciat- 
ed by those skilled in the art that other types of computer 
readable media that may store data that is accessible 
by a computer, such as magnetic cassettes, flash mem- 
ory cards, digital video disks, Bernoulli cartridges, ran- 
dom access memories (RAMs), read only memories 
(ROMs), and the like, may also be used in the example 
operating environment. 

[0032] A number of program modules may be stored 
on the hard disk drive 170, magnetic disk 190, optical 
disk 1 92, ROM 1 40, or RAM 1 50, including an operating 
system 1 95, one or more application programs 1 96, oth- 
er program modules 1 97, and program data 1 98. A user 
may enter commands and information into the computer 
1 00 through input devices, such as a keyboard 1 01 and 
a pointing device 1 02. Other input devices (not shown) 
may include a microphone, joystick, game pad, satellite 
dish, scanner, or the like. These and other input devices 
often are connected to the processing unit 110 through 
a serial port interface 1 06 that is coupled to the system 
bus 1 30, but may be connected by other interfaces, such 
as a parallel port, game port, or a universal serial bus 
(USB). Further still, these devices may be coupled di- 
rectly to the system bus 1 30 via an appropriate interface 
(not shown). A monitor 107 or other type of display de- 
vice is also connected to the system bus 130 via an in- 
terface, such as a video adapter 108. In addition to the 
monitor 1 07, personal computers typically include other 
peripheral output devices (not shown), such as speak- 
ers and printers. As one example, a pen digitizer 165 
and accompanying pen or user input device 1 66 are pro- 
vided in order to digitally capture freehand input. The 
pen digitizer 1 65 may be coupled to the processing unit 
1 1 0 via the serial port interface 1 06 and the system bus 
130, as shown in Fig. 1 , or through any other suitable 
connection. Furthermore, although the digitizer 165 is 
shown apart from the monitor 1 07, the usable input area 
of the digitizer 1 65 may be co-extensive with the display 
area of the monitor 107. Further still, the digitizer 165 
may be integrated in the monitor 107, or may exist as a 
separate device overlaying or otherwise appended to 
the monitor 107. 

[0033] The computer 1 00 may operate in a networked 
environment using logical connections to one or more 
remote computers, such as a remote computer 1 09. The 
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remote computer 109 may be a server, a router, a net- 
work PC, a peer device, or other common network node, 
and typically includes many or all of the elements de- 
scribed above relative to the computer 1 00, although on- 
ly a memory storage device 1 1 1 with related applications 
programs 1 96 have been illustrated in Fig. 1 . The logical 
connections depicted in Fig. 1 include a local area net- 
work (LAN) 112 and a wide area network (WAN) 113. 
Such networking environments are commonplace in of- 
fices, enterprise-wide computer networks, intranets, 
and the Internet, using wired and wireless communica- 
tion systems. 

[0034] When used in a LAN networking environment, 
the computer 1 00 is connected to the local network 1 1 2 
through a network interface or adapter 114. When used 
in a WAN networking environment, the personal com- 
puter 100 typically includes a modem 115 or other 
means for establishing a communications link over the 
wide area network 1 1 3, e.g., to the Internet. The modem 
115, which may be internal or external, is connected to 
the system bus 130 via the serial port interface 106. In 
a networked environment, program modules depicted 
relative to the personal computer 1 00, or portions there- 
of, may be stored in a remote memory storage device. 
[0035] It will be appreciated that the network connec- 
tions shown are exemplary and other techniques for es- 
tablishing a communications link between the comput- 
ers may be used. The existence of any of various well- 
known protocols such as TCP/IP, Ethernet, FTP, HTTP 
and the like is presumed, and the system may be oper- 
ated in a client-server configuration to permit a user to 
retrieve web pages from a web-based server. Any of var- 
ious conventional web browsers may be used to display 
and manipulate data on web pages. 
[0036] Fig. 2 illustrates an example of a pen-based or 
stylus-based computing system 201 that may be used 
in conjunction with various aspects of the present inven- 
tion. Any or all of the features, subsystems, and func- 
tions in the system of Fig. 1 may be included in the com- 
puter of Fig. 2. Pen-based computing system 201 in- 
cludes a large display surface 202, e.g., a digitizing flat 
panel display, such as a liquid crystal display (LCD) 
screen, on which a plurality of windows 203 is displayed. 
Using stylus 204, a user may select, highlight, and/or 
write on the digitizing display surface 202. Examples of 
suitable digitizing display surfaces 202 include electro- 
magnetic pen digitizers, such as Mutoh or Wacom pen 
digitizers. Other types of pen digitizers, e.g., optical dig- 
itizers, may also be used. Pen-based computing system 
201 interprets gestures made using stylus 204 in order 
to manipulate data, enter text, create drawings, and/or 
execute conventional computer application tasks, such 
as spreadsheets, word processing programs, and the 
like. 

[0037] The stylus 204 may be equipped with one or 
more buttons or other features to augment its selection 
capabilities. In one example, the stylus 204 may be im- 
plemented as a "pencil" or "pen," in which one end con- 



stitutes a writing element and the other end constitutes 
an "eraser" end. When moved across the display as an 
eraser, the eraser indicates portions of the display to be 
erased. Other types of input devices, such as a mouse, 
5 trackball, or the like, also may be used. Additionally, a 
user's own finger may be the stylus 204 and used for 
selecting or indicating portions of the displayed image 
on a touch-sensitive or proximity-sensitive display. Con- 
sequently, the term "user input device," as used herein, 
10 is intended to have a broad definition and encompasses 
many variations on well-known input devices, such as 
the stylus 204. Region 205 shows a feedback region or 
contact region permitting the user to determine where 
the stylus 204 contacted the display surface 202. 
15 [0038] An application program interface and systems 
and methods according to examples of this invention 
may be used with pen-based computing systems that 
accept and process electronic ink and ink strokes, like 
those described above in conjunction with Fig. 2. 

20 

III. GENERAL BACKGROUND ON INK LAYOUT 
ANALYSIS AND CLASSIFICATION ANALYSIS 

A. General Description of an Overall Ink Analysis 
25 System and Method 

[0039] To aid in understanding the present invention, 
it is useful to review some background information on 
ink "layout analysis" and ink classification analysis (also 

30 called "ink parsing"). While any suitable data processing 
systems and methods may be used without departing 
from this invention, in some examples of the invention 
layout analysis systems and methods like those de- 
scribed in U.S. Patent Appln. No. 10/143,865, filed May 

35 14, 2002, may be used, and in some examples of the 
invention, classification analysis systems and methods 
like those described in U.S. Patent Appln. No. 
10/143,864, filed May 14, 2002, may be used. In gener- 
al, parsing of ink may take place in any suitable manner 

40 without departing from this invention. 

[0040] Fig. 3 is a schematic diagram that generally il- 
lustrates an example of an overall system and method 
in which ink may be parsed or divided in some examples 
of this invention. In the example of Fig. 3, incoming or 

45 input ink strokes 300 first are subjected to a layout anal- 
ysis procedure 302, which combines and parses the in- 
put ink strokes 300 into associated stroke sets, such as 
words, lines, paragraphs (or blocks), and/or other 
groupings 304. In general, the layout analysis method 

50 or system 302 ascertains certain information relating to 
the size and layout of ink strokes 300 on a page, and 
groups together certain strokes based on size, layout, 
etc. An example of such a system or method is de- 
scribed in more detail in conjunction with Figs. 5, 6A, 

55 and 6B. 

[0041] After layout analysis 302, the data may be in- 
troduced into a variety of additional ink analysis engines. 
In the example system illustrated in Fig. 3, the data is 
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next introduced to a classification analysis system or en- 
gine 306. This classification analysis system or engine 
306 determines the type(s) of strokes included in the 
specific input data (e.g., whether individual strokes or 
stroke sets represent flow diagrams, freeform drawings, 
handwritten text, music, mathematics, charts, graphs, 
etc.). In some examples of the invention, if desired, a 
user may "inform" the system as to the type of input 
strokes, e.g., by selecting a "drawing mode," a "text 
mode," or the like, or by assigning a specific stroke type 
to one or more strokes (e.g., using a block or lasso select 
and assign procedure). 

[0042] Further processing of the input ink may depend 
on the stroke type recognized by the classification anal- 
ysis system or engine 306 (or otherwise determined). 
For example, for strokes or stroke sets that are classified 
as textual handwriting, the so-classified stroke sets may 
be sent to a handwriting recognition system 310 or an- 
other appropriate processing system. If necessary or 
desired, prior to introduction into the handwriting recog- 
nition system 31 0 or other processing system, the input 
ink data may be "normalized" using a normalization al- 
gorithm or system 308, to place the input ink data in an 
optimum orientation for analysis by the handwriting rec- 
ognition system 310 or other processing system (e.g., 
to rotate slanted input text strokes to a horizontal base 
line, if necessary). Conventional normalization systems 
or methods 308 and/or handwriting recognition systems 
or methods 310 may be used (if necessary and/or de- 
sired) without departing from the present invention. The 
data output from the handwriting recognition system or 
method 31 0 may constitute or link to machine-generat- 
ed text (e.g., lines, words, paragraphs, etc.) usable in 
any conventional manner, such as in conventional word 
processing systems (e.g., Microsoft WORD® or the 
like), e-mail handling systems, calendars, appointment 
books, etc. 

[0043] As another example, as illustrated in Fig. 3, if 
the classification analysis engine 306 recognizes the in- 
put strokes or stroke sets as containing drawing strokes, 
this data may then be transferred to an annotation rec- 
ognition system or method 314, which can be used, for 
example, to recognize textual information in the draw- 
ing. Further processing can proceed in any suitable 
manner. For example, if desired, the drawings may be 
"cleaned-up," wherein the handwritten annotations may 
be replaced with machine-generated text, handwritten 
drawing lines or shapes (e.g., circles, triangles, rectan- 
gles, etc.) may be replaced with machine-generated el- 
ements, and the like. Also, the drawings (either the 
handwritten versions or later machine-generated ver- 
sions) can be introduced into any suitable programs or 
systems without departing from this invention. 
[0044] The classification analysis systems and meth- 
ods 306 used in some examples of the invention also 
may recognize other specific writing or drawing types 
without departing from the invention. For example, a 
classification analysis system may recognize input 



stroke sets as containing music notations, mathematical 
information (such as formulas, mathematical symbols 
(+, -, =, %, x, sin, cos, tan, etc.), and the like), tables, 
charts, graphs, flow diagrams, schematic diagrams, 

5 drawings, sketches, doodles, etc., without departing 
from the invention. Such stroke sets, if present, could 
be sent to more specialized recognition systems and/or 
to other suitable processing applications without depart- 
ing from the present invention. 

w [0045] Some or all of the functions described in con- 
junction with Fig. 3 could be performed on input ink data 
after a user completely enters all ink onto the electronic 
page or document (e.g., upon a user's command, such 
as a "save," "parse," "close," or "recognize" command). 

15 Because of the computer processing time required to 
perform typical layout analyses and handwriting recog- 
nition analyses, however, a user may experience signif- 
icant delays if processing were conducted on this very 
infrequent, ad hoc basis. These processing delays may 

20 last long enough such that the user would become frus- 
trated waiting for the computer system to complete its 
analyses before moving on to the next desired opera- 
tions (e.g., entering more ink, moving on to a new page, 
printing, opening a new document or application, etc.), 

25 particularly if the electronic document is long or contains 
a large volume of ink. 

[0046] Systems and methods according to at least 
some examples of the present invention allow pen- 
based computing systems to perform various analyses, 

30 such as layout analysis 302, classification analysis 306, 
handwriting recognition analysis 310, etc., incremental- 
ly, in real time, while users continue using the pen-based 
computing systems (e.g., to enter and/or modify the ink 
strokes on the page). Moreover, in some examples of 

35 the systems and methods according to the invention, the 
various parser engines operate in a background thread, 
on a "snapshot" of the application data structure, in or- 
der to minimize the time that the application data struc- 
ture is unavailable to the user for entering ink (the term 

40 "application data structure," as used herein, means a 
data structure used in connection with an application 
program). While any suitable incremental data analysis 
systems and methods may be used without departing 
from the invention, examples of suitable systems and 

45 methods are described in U.S. Patent Appln. No. 
10/143,804, filed May 14, 2002. 

B. Description of Example Systems and Methods for 
Layout Analysis and Classification 

50 

[0047] Fig. 4 illustrates a schematic diagram of one 
example of a system useful for practicing the present 
invention. As illustrated, the overall system 41 0 includes 
an application system or program 420, which includes 
55 or communicates with a parser 422. The overall system 
410 may be embodied in a pen-based computing sys- 
tem like that illustrated in Fig. 2. The user 400 enters ink 
strokes into the system 41 0 (or the ink strokes are down- 
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loaded, e.g., from memory or an external source), and 
the ink strokes are stored by the application program 
420, for example, in an application data structure 402 
(which may be thought of as a document tree data struc- 
ture 402, like those illustrated in Figs. 6A and 6B). So 
that the user 400 can continue to make changes to the 
document tree data structure 402 while the parser 422 
operates, the parser 422 contains a mirror tree data 
structure 404. Changes made to the document tree data 
structure 402 (e.g., by the user 400, the parser 422, from 
another source, etc.) are immediately passed on to the 
mirror tree data structure 404 so that the mirror tree data 
structure 404 generally "mirrors" the content of the doc- 
ument tree data structure 402. 

[0048] The mirror tree data structure 404 is used to 
supply input data to the two analysis engines 406 and 
408 in the parser 422. In the example illustrated in Fig. 
4, one analysis engine is a layout analysis engine 406 
(which may conduct, for example, a layout analysis 302, 
as discussed above in conjunction with Fig. 3), and the 
other is a recognition engine 408 (which may conduct, 
for example, handwriting recognition analysis 310 and/ 
or annotation recognition analysis 314, as discussed 
above in conjunction with Fig. 3). The engines 406 and 
408 receive "snapshots" 424 and 426, respectively, of 
the mirror tree data structure 404 as input data, and they 
operate on these "snapshots" 424 and 426 in back- 
ground instead of operating directly on the document 
tree data structure 402 or the mirror tree data structure 
404. In this manner, the user 400 can continue perform- 
ing operations on the document tree data structure 402 
in the application program 420 {e.g., adding ink, deleting 
ink, modifying ink, etc.) while the various parser analysis 
engines 406 and 408 also are operating, and the user 
400 does not experience significant interruptions in op- 
eration (e.g., processing delays) as the engines 406 and 
408 operate on the data. 

[0049] To produce "snapshots" 424 and 426 in some 
examples, existing snapshot data structures may be 
compared with the mirror tree data structure 404. The 
differences between the two are noted, and a minimal 
number of operations are performed to synchronize the 
snapshot 424 or 426 to the mirror tree data structure 
404. In this manner, minimal data rewrite occurs in mak- 
ing the snapshot (e.g., unchanged data from a previous 
snapshot is not rewritten), which also helps speed up 
operation of the parser 422. 

[0050] The output of the parser engines 406 and 408 
may be modified or revised data structures. For exam- 
ple, if the layout analysis engine 406 is like that illustrat- 
ed in Fig. 5, the output of layout analysis engine 406 
may be a data structure that includes individual ink 
strokes grouped into associated words, lines, para- 
graphs, and the like. Operation of a layout analysis en- 
gine of this type is described in more detail below. Sim- 
ilarly, if the parser engine 408 is a handwriting recogni- 
tion system 310, the output may include information or 
a data structure that ties the ink strokes to machine-gen- 



erated text. 

[0051] When the parser engines 406 and 408 com- 
plete their operations on the snapshot input data 424 
and 426, respectively, the resulting information may be 
5 sent back to the application program 420, as indicated 
by arrows 428 and 430, respectively. As noted above, 
however, the user 400 may change the document tree 
data structure 402 during the time period that the parser 
engines 406 and 408 operate on the snapshots 424 and 
10 426. Therefore, before writing the parser analysis en- 
gine results back to the document tree data structure 
402, the parser 422 compares the document tree data 
structure 402 currently in the application program 420 
(including the user's changes) to the revised document 
15 tree data structure(s) sent by the parser engines 406 
and 408, optionally using the mirror tree data structure 
404. If the user 400 made changes to the document tree 
data structure 402 that are not contained in the revised 
document tree datastructure(s) from the parser engines 
20 406 and 408 (e.g., by adding, deleting, moving, resizing, 
or otherwise modifying one or more strokes), or if user- 
made changes to the document tree data structure 402 
render moot or conflict with changes to the data struc- 
ture^) made by the parser engines 406 and 408 (e.g., 
25 by adding, deleting, or otherwise modifying strokes), 
then the application document tree data structure 402 
is revised only to include the changes made by the pars- 
er analysis engines that do not conflict with the user- 
made changes (user-made changes will override pars- 
30 er-made changes). Also, only portions of the document 
tree data structure 402 modified from the existing ver- 
sion are changed or rewritten, in order to reduce data 
writing time (and the associated interruption experi- 
enced by the user 400). In this manner, the finally re- 
35 vised document tree data structure 402 present in the 
application program 420 will include all changes made 
by the user 400 and the results of the previous parser 
engine analyses, to the extent that the parser engine 
made changes that are not inconsistent with or trumped 
40 by user-made changes. 

[0052] Because the document tree data structure 402 
contains shared data ultimately modifiable by the user 
400 as well as the parser engines 406 and 408, the user 
400 cannot input new data into the document tree data 
45 structure 402 while it is being rewritten to include the 
parser-made changes. If a user 400 attempts to do so, 
systems and methods according to the invention can 
handle these efforts in any suitable manner. For exam- 
ple, the new strokes or changes may be ignored, or they 
50 may be stored in a temporary buffer memory until the 
revised application document tree data structure 402 is 
available for data input. However, because the docu- 
ment tree data structure 402 in the application program 
420 according to this example of the invention generally 
55 is unavailable only during the time the system rewrites 
the changed portions of the data structure 402, the un- 
available time period typically is quite short, and often 
unnoticed by the user. 
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[0053] Once the document tree data structure 402 is 
rewritten or modified (including the user and/or parser 
engine made changes), the mirror tree data structure 
404 is updated to mirror the rewritten or modified docu- 
ment tree data structure 402, and the parser engines 
406 and 408 can repeat their analyses (if necessary). 
Advantageously, in some examples, the parser engines 
406 and 408 will operate only on the portions of the doc- 
ument tree data structure that have been recently mod- 
ified (and any portions affected by the recent modifica- 
tions), to reduce processing time. By incrementally up- 
dating the parser engine operations at the same time 
the user inputs data, the parser 422 can generally keep 
up with the user's data entry, thereby minimizing 
processing delays observed by the user. 
[0054] As mentioned above, in some examples of the 
invention, processing time may be reduced by limiting 
processing to portions of the data structure where 
changes have occurred (and all areas affected by these 
changes). If user input or previous parser engine oper- 
ations have not affected some portions of a data struc- 
ture, there may be no need for the parser engine(s) to 
again analyze these same portions (and presumably ar- 
rive at the same results). As examples, systems and 
methods according to some examples may reanalyze 
any portion of the data structure located within a prede- 
termined distance of a change. For example, reanalysis 
may include the line of any change and any one or two 
lines surrounding the change, any strokes located within 
a circle of a pre-selected radius surrounding the change, 
any block of text (as described in more detail below) in- 
cluding a change, or the like. The following explains ex- 
amples of parsers that take advantage of these features 
in more detail. 

C. An Example of Processing Taking Place During 
Parsing 

[0055] The data analyzed or processed in systems 
and methods according to examples of the present in- 
vention can take on any suitable form or structure. For 
example, in one procedure as illustrated in Fig. 3, indi- 
vidual strokes 300 of input ink data are combined to- 
gether into a data structure as a result of a succession 
of decisions made by a layout analysis engine 302, 
which groups or associates certain individual strokes 
based on overall ink layout and statistics obtained from 
the input ink. The layout analysis engine 302 may pro- 
vide a hierarchical grouping of ink strokes on a page, 
which allows global statistic calculations over the group 
(s). The first stroke grouping decisions are conservative, 
based on local layout relationships when the groups of 
ink strokes are small (e.g., small groups representing 
individual ink strokes or relatively short combinations of 
strokes). Later stroke grouping decisions can be more 
aggressive, due to the larger statistic sample size col- 
lected from the larger ink stroke groupings (e.g., stroke 
sizes over a longer line, relative stroke spacing, line an- 



gles, etc.). Multiple passes through the input ink data 
may be conducted to enable increasingly aggressive 
decision making in determining whether to merge 
strokes to form stroke sets, such as words, lines, and/ 

5 or blocks 304 of input ink strokes. 

[0056] Fig. 5 generally illustrates steps or parse en- 
gines involved in one example of an ink layout analysis 
parser engine, system, or method 302 useful in produc- 
ing and/or modifying data structures used in some ex- 

10 amples of this invention. Because of the very high de- 
gree of freedom provided to users in inputting digital ink 
into systems and methods according to some examples 
of the invention (e.g., a user is allowed to write anywhere 
on a digitizer input screen, in any orientation, at any 

15 time, using any desired stroke size), when the layout 
analysis procedure 302 of Fig. 5 begins, there may be 
no preliminary information available from which to de- 
termine the proper layout, orientation, or type of input 
data {e.g., whether the incoming input data 500 is tex- 

20 tual, drawing, mathematic, music, flow diagrams, 
charts, graphs, etc.). Element 502 in Fig. 5 provides a 
general graphical representation of one of the types of 
possible input data structures 500 at the start of this lay- 
out analysis procedure. The graphical representation 

25 502 is illustrated in more detail in the parse tree data 
structure of Fig. 6A. In general, when the layout analysis 
procedure 302 begins (e.g., even as the user may con- 
tinue to input ink strokes into the pen-based computing 
system), the system treats every stroke S 600 on a given 

30 page (or in a given document) P 608 as a separate word 
W 602, every word W 602 is treated as a separate line 
L 604, and every line L 604 is treated as a separate block 
B 606 (or paragraph). The layout analysis engine 302 
performs the task of associating or merging strokes to- 
ss gether to form stroke sets containing proper words, 
lines, and blocks of associated ink data. While any suit- 
able layout analysis engine could be used in conjunction 
with this invention, the example illustrated in Fig. 5 is 
described in more detail below. 

40 [0057] While this description of the exemplified layout 
analysis engine 302 uses terms like "word," "line," and 
"block," these terms are used in this portion of the spec- 
ification as a matter of convenience to refer to one or 
more associated strokes or stroke sets. At the time the 

45 layout analysis 302 initially occurs in at least some ex- 
amples of the invention, no final determination has been 
made as to whether individual strokes or stroke sets 
constitute writing, drawings, music, etc. Also, while the 
above description uses the term "page," it is not neces- 

50 sary that a given electronic document be parsed on a 
page-by-page basis. For example, "blocks" or "para- 
graphs" of electronic documents could bridge two or 
more pages of a document without departing from the 
invention. 

55 [0058] The layout analysis engine 302 according to 
this example of the invention operates greedily, such 
that during each pass (or operation of each parse en- 
gine), stroke or line merger operations occur, but splits 
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do not. Moreover, the engine 302 may be operated with 
appropriate tests and tolerances such that it should not 
be necessary to go back and correct an undesired merg- 
er operation. 

[0059] As a result of the operation of layout analysis 
engine 302, the individual strokes 600 of an electronic 
document may be combined together into associated 
stroke sets including words W, lines L, and blocks B (or 
paragraphs), where appropriate. Fig. 6B illustrates a 
graphical representation 506 of a possible data struc- 
ture for the data output 504 from a layout analysis en- 
gine 302. As evident from a comparison of Figs. 6A and 
6B, the Page (or document) 608 overall contains the 
same stroke information, but certain strokes S 600 have 
been combined or associated together to form words W 
610, and certain words W 610 have been joined together 
to form a line L 612 in the data structure of Fig. 6B. Of 
course, a word W 610 may contain any number of 
strokes S 600, and likewise a line L 612 may contain 
any number of words W 61 0. Also, although not illustrat- 
ed in the particular parse tree example of Fig. 6B, two 
or more lines L 61 2 also may be joined together to form 
a block B 614 (or paragraph). 

[0060] In addition to helping define the structure of ink 
in a document, the various nodes in the parse tree (e. 
g., nodes 600, 61 0, 61 2, etc. in Fig. 6B) may be used to 
store spatial information relating to various levels in the 
tree. For example, each line level node 612 may store 
a regression/fit line of all points that make up the strokes 
of the line, the convex hull of each stroke in the line, and/ 
or any other desired information. Also, the parse tree 
data structures can be modified by applying various el- 
ementary operations on the strokes, words, lines, and 
blocks contained in it. Suitable operations may include: 
add, remove, merge, split, and re-parent. More complex 
operations may be composed using these elementary 
operations. As these operations are performed on the 
data structure tree, the statistics maintained at the dif- 
ferent node levels may be automatically updated to cor- 
respond to the new structure. 

[0061] Fig. 5 provides a schematic overview of one 
example of a suitable layout analysis engine 302 useful 
in some examples of this present invention. In this ex- 
ample, a first step in the layout analysis procedure 302 
is a temporal line-grouping step 508, which generally 
compares features of temporally adjacent strokes (i.e., 
consecutively written strokes) and combines them as 
"lines," if appropriate. Various factors may be taken into 
account in determining whether atemporal line grouping 
should be made from two or more temporally adjacent 
strokes, such as stroke size, inter-stroke spacing, stroke 
angle, etc. Once this temporal line grouping step 508 is 
completed, the next step in the analysis 302, a spatial 
block grouping step 51 0, compares the physically adja- 
cent temporal line groupings formed in step 508 and 
combines the temporal line groupings that are located 
close to one another as spatial blocks. Various factors 
may be taken into account in determining whether aspa- 



tial block grouping should be made from adjacent tem- 
poral line groupings, such as stroke size, inter-stroke 
spacing, line angle, etc. 

[0062] The temporally grouped lines (from step 508) 
5 may be further grouped if appropriate, optionally taking 
into consideration their spatial block relationship or ori- 
entation, in a spatial line grouping step 512. This spatial 
line grouping step 51 2 need not consider the time of one 
stroke compared to another stroke, although factors in 
10 addition to the lines' spatial relationship may be taken 
into consideration, such as line angle, stroke size, etc. 
Also, the results of the spatial block grouping procedure 
510 described above may be used as a factor in deter- 
mining whether a spatial line grouping should be made 
15 between two existing temporal line groupings. 

[0063] Once the spatial line groupings have been 
completed, the layout analysis procedure 302 according 
to this example may then combine the individual strokes 
in the line groupings into one or more spatial word 
20 groupings 51 6, depending, for example, on factors such 
as inter-stroke spacing, line orientation, stroke size, etc. 
The resulting output 504 may be a data structure 506 
with strokes grouped into words, lines, and blocks, as 
explained in conjunction with Fig. 6B. 
25 [0064] Fig. 5 also illustrates an optional parse engine 
or step in broken lines that may be utilized as part of a 
layout analysis 302. This optional step is called "list de- 
tection" 514. Often, when people write a list, they tend 
to write a (vertical) column of numbers, letters, or bullet 
30 points, and then fill in the list elements (in the horizontal 
direction). At other times, people will write out the con- 
tent of the list, and then later add a vertical column of 
numbers, letters, or bullet points. The list detection en- 
gine 514 may detect these special circumstances (e.g., 
35 by looking at the orientation and timing of temporal line 
groupings, etc.) and combines the list number, letter, or 
bullet point strokes with the corresponding list element 
text. 

[0065] The various steps in this exemplified ink anal- 
40 ysis engine 302 (Fig. 5) may be changed in order or 
omitted without departing from the invention. For exam- 
ple, if desired, the spatial line-grouping step 512 may 
take place before the spatial block-grouping step 510. 
[0066] The output data 504 from the layout analysis 
45 engine 302 can be used in any suitable manner, such 
as in a classification engine 306, as illustrated in Fig. 3, 
and from there the data may proceed to other appropri- 
ate processing engines (e.g., annotation recognition 
314, handwriting recognition 310, etc.). Layout analysis 
50 engine 302, or a combination of layout analysis engine 
302 and classification engine 306, may form a parser 
engine 406 as illustrated in conjunction with Fig. 4. 
[0067] Of course, the present invention is not limited 
to operation with a layout analysis engine or any specific 
55 type of analysis engine. Other suitable engines or pro- 
cedures for grouping or associating individual strokes 
into appropriate data structures or any other desired 
analysis can be performed without departing from this 
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invention. Also, if desired, prior to processing, the user 
could indicate to the system that certain strokes always 
should be grouped together (e.g., by drawing a line or 
lasso around, highlighting, or otherwise selecting input 
data strokes to be associated together). 

IV. AN INK DIVIDER OBJECT AND API 

A. General Description 

[0068] This specification continues with a detailed de- 
scription of examples of parsers and application pro- 
gramming interfaces according to the invention, includ- 
ing a specific example, namely the InkDivider object. 
One function of the systems and methods according to 
the invention {e.g., the InkDivider object) is to take a col- 
lection of ink strokes provided by an application and di- 
vide these strokes into parsed entities of specified gran- 
ularity (e.g., into words, lines, sentences, paragraphs, 
drawings, or the like). Without proper parsing, electronic 
ink either tends to become far too granular (i.e., having 
large numbers of ungrouped strokes), or it tends to be- 
come grouped together as a single ink object, making 
desired moving, selecting, scaling, and other opera- 
tions, particularly of individual ink strokes or small 
groups of ink strokes, difficult or impossible. The sys- 
tems and methods according to the invention, including, 
for example, an ink divider object and API, exposes the 
parsing technology and results to the developer com- 
munity, which thereby allows code writers to take advan- 
tage of and use the parsing engine results when writing 
code for new applications. 

[0069] In general, during an inking session, in some 
manner, ink strokes will be added to and/or deleted from 
a collection of ink strokes present in the application. Ad- 
ditionally, during an inking session, existing strokes 
within the ink stroke collection may be moved, resized, 
partially erased, and/or otherwise modified. 
[0070] When an inking session ends (and optionally 
incrementally while an inking session is taking place), 
the operating application program will call the parser (e. 
g., included in the InkDivider object), which processes 
the strokes into stroke sets or groups of different gran- 
ularity (at least the new strokes and/or changed strokes 
and/or any strokes affected by the new and/or changed 
strokes since a previous call of the parser). In general, 
when the parser is called, the application program sup- 
plies ink strokes to the parser and receives back certain 
information. In some examples, the information returned 
contains a back-pointer that identifies the original 
strokes that were divided. Systems and methods ac- 
cording to some examples of the invention also may pro- 
vide a method (called "ResultByType" in this example) 
for retrieving a desired collection of strokes of specified 
granularity. For example, the application program can 
query the division results to obtain units of different pars- 
ing granularity, depending on the desired granularity 
type (e.g., words, lines, blocks, drawings, etc.). The 



parsing results also may have a notion of baseline for 
the purpose of correcting angled writing to a horizontal 
baseline before feeding this data to a handwriting rec- 
ognizer, if necessary and/or desired. This may be ac- 
5 complished, for example, by making the rotation matrix 
available to code writers. 

[0071] Notably, individual ink strokes may belong to 
multiple ink stroke collections or different granularity 
groupings, e.g., a stroke can be part of a Word and part 

10 of a Paragraph. 

[0072] With this general background and overview in 
mind, the various features of an ink divider object and 
API examples according to the invention are discussed 
in more detail below. While much of the following dis- 

15 cussion relates to a specific ink divider object and its 
associated objects, properties, etc., those skilled in the 
art will recognize that various modifications can be 
made to the specific implementations described below 
without departing from the invention. 

20 

B. Ink Divider Object 

[0073] Fig. 7 generally illustrates the content of an ex- 
ample InkDivider object 700 useful in some examples 

25 of the invention. In this example, the InkDivider object 
700 contains two properties 702, namely a Strokes 
property 704 and a RecognizerContext property 706. 
This illustrated InkDivider object 700 also includes one 
method 708, namely a Divide method 710. 

30 [0074] The Strokes property 704 returns and/or sets 
the collection of ink strokes to be subjected to the Divide 
method 710. The strokes generally are sent to the 
Strokes property 704 by the application program being 
used, which determines which strokes to add to and/or 

35 remove from and/or otherwise modify in the collection 
of strokes in the Strokes property 704. This generally is 
shown in Fig. 7 by arrow 712, which represents incom- 
ing strokes sent by the application program (or other- 
wise sent to the Strokes property 704 in any suitable 

40 and/or desired manner). Strokes also may be added to 
and/or removed from the Strokes property 704 by the 
ultimate user in any suitable manner, such as through 
an ink insertion action, an ink paste action, an ink cut or 
delete action, etc. If desired, the stroke information sent 

45 to the Strokes property 704 need not include all features 
or properties of the ink strokes, such as color. Rather, if 
desired, it is sufficient to send only the features or prop- 
erties of the ink strokes relevant to parsing. 
[0075] The input and output data for the Strokes prop- 

50 erty 704 may take on the following form: 

[propputrefjHRESULT Strokes([in]lnkStrokes* 

Strokes); 

55 [propgetjHRESULT Strokes([out,retval]lnk- 
Strokes** Strokes). 

[0076] The RecognizerContext property 706 returns 
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and/or sets the recognizer to be used and/or associated 
with the InkDivider object 700. This property 706 is use- 
ful in at least some examples of the invention because 
the desired parsing operation may include handwritten 
text recognition to be based on a language different from 
the default language of the system on which the soft- 
ware or application program is being run. For example, 
a pen-based computing system may have English as 
the default language for its operating system and/or key- 
board. If the computer user is bilingual or if another user 
uses the computer, however, in some instances, the us- 
er may write or take notes in a language other than Eng- 
lish. If a default English language handwriting recogniz- 
er is the only option available on the system, this may 
result in errors as the recognizer attempts to recognize 
the non-English handwritten text. Other specialized rec- 
ognizers also could be set by application code, for ex- 
ample, specialized recognizers for recognizing musical 
notations, mathematical formulas and symbols, drawing 
features, etc. By enabling code writers to set and/or use 
different handwriting recognizers (including recognizers 
for different languages), the resulting handwriting rec- 
ognition results may be improved. The ability for code 
writers to set a desired handwriting recognizer is illus- 
trated in Fig. 7 by arrow 714. 

[0077] In some examples of the invention, the Recog- 
nizerContext property 706 may default to a "null" value, 
which, in these examples, means that the operating sys- 
tem default language of the computer will be used as 
the handwriting recognition language and/or a recogniz- 
er supplied with the operating system will be used as 
the recognizer unless and until the RecognizerContext 
property 706 is changed to specify another recognizer. 
The default or "null" language value may correspond to 
the "keyboard" default locale ID set during initial system 
set up. This default or null input feature is illustrated in 
Fig. 7 by arrow 716, labeled "NULL." 
[0078] The input and output data for the Recog- 
nizerContext property 706 may take the following forms: 

[propputref]Recognizer([in]lnkRecognizer* Rec- 
ognizer); 

[propget]Recognizer([out,reval]lnkRecognizer** 

Recognizer). 

[0079] In operation, in at least some examples of the 
invention, a parser will make a first pass at determining 
word breaks in the handwritten text based on spatial and 
temporal metadata associated with the ink strokes. This 
may include, for example, the temporal line grouping, 
spatial block grouping, and spatial line grouping steps 
generally described above in conjunction with Fig. 5. 
These parsing operations can be conducted relatively 
quickly, but generally the results will not include the so- 
phistication and accuracy associated with a handwriting 
recognizer and its associated language model. There- 
fore, in at least some examples of the invention, a hand- 
writing recognizer is used to make a second pass at 



each handwritten "line" of text and accurately identify 
breaks between the words, using the dictionary associ- 
ated with the handwriting recognizer to better identify 
word breaks. The recognizer also can convert the hand- 

5 written text into machine-generated text (e.g., into a for- 
mat suitable for a word processing program, email, elec- 
tronic calendars and appointment books, etc.). 
[0080] Fig. 7 also illustrates a Divide method 710, 
which constitutes part of the InkDividerobject 700 in this 

10 example. This method 710 divides or parses the asso- 
ciated strokes (obtained from the Strokes property 704 
- see arrow 71 8) using the recognizer set by the Recog- 
nizerContext property 706 (see arrow 720). The Divide 
method 71 0 produces and returns an InkDivisionResult 

15 object 800 that contains the results of the ink division 
(or parsing). The generation and return of the InkDivi- 
sionResult object 800 is illustrated in Fig. 7 by arrow 
722. An example of the available output data format of 
the Divide method 71 0 is shown below: 

20 

HRESULT Divide ([out, retval] InkDivisionResult** 
divisionResults). 

[0081 ] In at least some examples of the invention, the 
25 Divide method 71 0 is performed or called synchronous- 
ly, while additional ink may be added to, deleted from, 
or otherwise modified in the document in the application 
program. In additional examples of systems and meth- 
ods according to the invention, the Divide method 710 
30 may operate in a background thread on the strokes pro- 
vided via the Strokes property 704, and it does not return 
an InkDivisionResult 800 until the entire parsing opera- 
tion is completed. By operating in a background thread 
and without affecting further stroke entry or modification, 
35 in many instances the use of Divide method 71 0 will be 
transparent or almost transparent to the pen-based 
computing system user and will not cause significant 
processing delay. 

[0082] Each time the Divide method 710 is called, a 

40 new InkDivisionResult object 800 may be created, 
which effectively captures a snapshot of the ink parse 
tree data structure (see Fig. 6B) at the time the Divide 
method 710 is called. In at least some examples of the 
invention, it is the responsibility of the application pro- 

45 gram to compare the Strokes of each InkDivisionResult 
object 800 (discussed in more detail below) to determine 
whether the parsing results have changed between dif- 
ferent calls of the Divide method 710. 
[0083] Of course, without departing from the inven- 

50 tion, an ink divider object may include methods, proper- 
ties, and/or other elements in addition to and/or in place 
of and/or in combination with the specific methods and 
properties illustrated in Fig. 7. As one example, an ink 
divider object additionally may include a "Line Height" 

55 property (alternatively, the Line Height property could be 
associated with another object or provided in any suita- 
ble manner). The Line Height property allows a code 
writer, as input, to set an expected Line Height for lines 
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of text. In this manner, during parsing, the layout analy- 
sis system and/or classification analysis system (or oth- 
er systems in the parser) can use this expected line 
height information and do a better job in differentiating 
between lines of text and drawings or diagrams. By giv- 
ing the parser this line height guideline, it can more ef- 
fectively and efficiently differentiate multiple lines (e.g., 
in a paragraph orientation) or single lines from drawings 
(e.g., drawing lines are typically taller than a single line 
of handwriting). 

[0084] While no limit on the expected line height size 
of handwritten lines is necessary, in some examples of 
the invention, the systems and methods will accept ex- 
pected line heights that fall within a certain range. Of 
course, this range of expected line heights may vary 
widely. In some examples of the invention, the expected 
line height must fall within a minimum height of 1 00 dig- 
itizer pixels and a maximum height of 50,000 digitizer 
pixels, with a default height of 1 200 pixels. If a code writ- 
er attempts to set an expected line height outside of 
these ranges, the Line Height property may return an 
error message. Alternatively, the Line Height Property 
may automatically change the input line height to the 
relevant minimum or maximum height value without re- 
turning an error message (e.g., automatically setting a 
line height value of 50,000 if an application program 
code attempts to set the value at 50,003). As another 
alternative, attempted setting a line height value outside 
the valid range may result in the value simply being ig- 
nored (and reverting back to the previous line height val- 
ue or the default value). 

[0085] As output, the Line Height property will tell the 
application program the previously set value for the Line 
Height property, or it will return the default value if no 
previous value had been set. 

[0086] As an example, the input and output data for 
the Line Height property according to this example of 
the invention may take on the following forms: 

HRESULT [propput] LineHeight([in]Long Line- 
Height); 

HRESULT [propget] LineHeight([out, retvaljLong* 
LineHeight). 

C. Ink Division Result Object 

[0087] Fig. 8 graphically illustrates an InkDivisionRe- 
sult object 800 according to some examples of the in- 
vention. As noted above, the Divide method 710 of the 
InkDivider object 700 parses the stroke collection (ob- 
tained from Strokes property 704) based on the selected 
RecognizerContext property 706 and creates an InkDi- 
vision Result object 800. The InkDivisionResult object 
800 captures the data structure resulting from the divi- 
sion and/or parsing operations, which may be consid- 
ered to be a "parse tree" of the form illustrated in Fig. 
6B, in at least some examples of the invention. The re- 



sulting data structure present in the InkDivisionResult 
object 800 can be further used, e.g., in subsequent Re- 
sultByType operations (described in more detail below), 
to retrieve ink data sets of different levels of granularity 

5 for a given stroke collection. 

[0088] As illustrated in Fig. 8, this example of the Ink- 
DivisionResult object 800 has a property 802 called 
"Strokes" 804. This Strokes property 804, when called, 
returns a reference to the strokes that were originally 

10 used in producing the InkDivisionResult 800. The Ink- 
Divider object 700 internally builds a data structure that 
corresponds to a specific set of strokes at an instant in 
time. The Ink object containing these strokes, however, 
is not static. Rather, new strokes can be added (either 

15 individually or in bulk, e.g., through a paste operation) 
and existing strokes can be deleted or moved or other- 
wise modified at any time (even while a parsing opera- 
tion is being carried out). Therefore, the Strokes prop- 
erty 804 in the InkDivisionResult object 800 provides ap- 

20 plication program code or client code a means of deter- 
mining: (a) which strokes were subject to division to cre- 
ate a particular InkDivisionResult object 800 and (b) 
whether those strokes have been affected or modified 
since the last InkDivisionResult object was obtained (e. 

25 g mi since the last Divide method 710 call). This Strokes 
property 804 also allows the application code or client 
code to compare two InkDivisionResult objects to deter- 
mine whether the parse tree has changed from one Di- 
vide method call to the next. 

30 [0089] The Strokes property 804 of the InkDivision- 
Result object 800 receives, contains, and/or maintains 
a list of strokes used in producing the InkDivisionResult 
object 800. This is illustrated in Fig. 8 by input arrow 
806. This input stroke data can be obtained or intro- 

35 duced, for example, from the Strokes property 704 of 
the InkDivider object 700, or from any other suitable 
source. The ability to output data representing the ink 
strokes used in obtaining the InkDivisionResult object 
800 is illustrated in Fig. 8 as arrow 808 from the Strokes 

40 property 804. The output data 808 of the Strokes prop- 
erty 804 may take the following form: 

[propg et] H R ES U LT Strokes ([out, retval] In k- 

Strokes** Strokes). 

45 

[0090] Because the InkDivider object 700 encapsu- 
lates the parsing engine and the InkDivisionResult ob- 
ject 800 encapsulates the parsing tree data structure for 
a specific ink division operation, it is possible to release 
50 the InkDivider object 700 (e.g., for further operations) 
while one or more InkDivisionResult objects 800 contin- 
ue to exist. 

[0091 ] As an alternative, rather than include a Strokes 
property 804 in the inkDivisionResult object 800, client 
55 code or application program code could cache the 
stroke information externally. However, with the likely 
creation of multiple InkDivisionResult objects 800 over 
the course of an inking session, it may be difficult and 
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computationally expensive to manage the pairs of Ink- 
DivisionResult objects 800 and the external ink stroke 
data sets. Therefore, providing the Strokes property 804 
as part of the InkDivisionResult object 800 reduces 
overhead for the client code or application code and aids 
in effectively utilizing the ink divider API. 
[0092] The InkDivisionResult object 800 according to 
this example further includes a method 810 called Re- 
sultByType 812, as further illustrated in Fig. 8. This Re- 
sultByType method 812, when called, returns the col- 
lection of strokes collections (in division Units) that re- 
sulted from a given DivisionType (the type of division, e. 
g., words, lines, paragraphs, drawings, etc.). As exam- 
ples, this method 812 returns the input strokes grouped 
as words, lines, paragraphs, drawings, etc., depending 
on whether the client code or application code request- 
ed words, lines, paragraphs, drawings, etc. This method 
812 can be called multiple times, if desired, to retrieve 
division results for various different parsing granulari- 
ties. For example, one call could provide the words in 
the parsed stroke collection, another call could provide 
the lines, another the paragraphs, etc. 
[0093] Input to the ResultByType method 812 in- 
cludes at least the InkDivisionType desired, which, as 
noted above, in some examples may mean words, lines, 
paragraphs, drawings, etc. This input is illustrated in Fig: 
8 by input arrow 81 4. The output, which includes the col- 
lection of stroke collections for the given DivisionType 
(e.g., an InkDivisionUnits object), is represented in Fig. 
8 by output arrow 81 6. This data may take the following 
format: 

HRESULT ResultByType ([in] InkDivisionType divi- 
sion Type, [out, retval] InkDivisionUnits* 
divisioni/nits ). 



methods and properties illustrated in Fig. 8. 
D. Ink Division Units Object 

5 [0097] Fig. 9 illustrates an InkDivisionUnits object 900 
useful in some examples of this invention. This object 
900 is a collection wrapper for the results of the parsing 
operation. The collection, in at least some examples of 
this invention, typically is expected to comprise effec- 

10 tively all of the strokes originally given to the InkDivider 
object 700. For example, a collection of strokes that has 
been divided into Words may be represented by an Ink- 
DivisionUnits object 900 collection that contains asingle 
InkDivisionUnit object 1 000 for each word (see also Fig. 

15 1 o, described in more detail below). The strokes that re- 
sult from expanding the individual InkDivisionUnit ob- 
jects 1 000 into their respective strokes would be expect- 
ed to match the original set of strokes passed to the Ink- 
Divider object 700. 

20 [0098] As illustrated in Fig. 9, the InkDivisionUnits ob- 
ject 900 of this example contains a property 902 called 
Count 904. The Count property 904 provides the count 
(or number) of division units present in a given stroke 
collection and makes this information available to the 

25 API for use in application code or client code. For ex- 
ample, the Count property 904 may be able to inform an 
application program that a given stroke collection con- 
tains x number of words and/or y number of lines and/ 
or z number of paragraphs. This data can be determined 

30 in any suitable manner, for example, by looking at the 
parse tree data structure of Fig. 6B for a given stroke 
collection, which exists in the InkDivisionResult object 
800. The output of the Count property 904 is illustrated 
in Fig. 9 by arrow 906. The output data may be struc- 

35 tured as follows: 



[0094] In some examples of the invention, if no Divi- 
sionType is specified (Division Type = NULL), as repre- 
sented in Fig. 8 by arrow 818, then the returned InkDi- 
visionUnits object collection, in at least some examples 
of the invention, may include all of the granularities iden- 
tified by the parser. Because some granularities sub- 
sume other granularities [e.g., a Line may contain sev- 
eral Words), the resulting collection may contain over- 
lapping division units. Of course, any suitable default 
granularity (or even no default granularity) can readily 
be used without departing from the invention. For exam- 
ple, in some versions, the default DivisionType may be 
"WORD" unless and until another granularity is speci- 
fied by the user code or application code. 
[0095] As illustrated in Fig. 8, the ResultByType meth- 
od 812 may receive the input data structure from the 
Divide method 71 0 of the InkDivider object 700, as illus- 
trated by arrow 820. 

[0096] Of course, without departing from the inven- 
tion, an inkDivisionResult object 800 may include meth- 
ods, properties, and/or other elements in addition to 
and/or in place of and/or in combination with the specific 



[propget] HRESULT Count([out,retval]Long** 

Count). 

40 [0099] The InkDivisionUnits object 900 of this exam- 
ple further includes a method 908 called Item 910. The 
Item method 910, when called, returns a specific InkDi- 
visionUnit object 1 000 in the collection of strokes given 
the unit's Index value in the collection (e.g., "return the 
45 fourth word"). The output is represented in Fig. 9 by ar- 
row 912. The output data of the Item method 910 may 
be structured as follows: 

HRESULT Item ([in] Long index, [out.retval] InkDi- 
50 vision Unit* division Unit). 

[0100] Another property 902 contained in the InkDivi- 
sionUnits object 900 of this example is called 
"_NewEnum" 914. This property 914 returns either the 
55 lEnum VARIANT or lEnum UNKNOWN enumerator in- 
terface for the stroke collection being evaluated. This 
property 91 4 may be used to retrieve any individual ob- 
ject in the ink stroke collection being evaluated, as illus- 
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trated in Fig. 9 by output arrows 916 and 918, respec- 
tively. The data evaluated by the_NewEnum method 
914 may take on the following format: 

[propget] HRESULT _NewEnum ([out, retvaljllln- 
known** _NewEnum). 

[0101] Notably, in this example of the invention, (a) 
the Count property 904 plus the Item method 910 and 
(b) the _NewEnum property 914 described above are 
effectively two equivalent ways of accessing the ele- 
ments of the ink stroke collection, depending on the pro- 
gramming language and coding style used by the client 
or application program code. The combination of Item 
method 91 0 and Count property 904 could be used in a 
traditional "for" loop, while the _NewEnum property 914 
could be utilized by the "for each" construct available in 
some programming languages. 

[0102] Of course, without departing from the inven- 
tion, an InkDivisionUnits object 900 may contain prop- 
erties, methods, and/or other elements in addition to 
and/or in place of and/or in combination with the specific 
properties and methods described above in conjunction 
with Fig. 9. 

E. Ink Division Unit Object 

[0103] Another object in the API according to some 
examples of the invention, as illustrated in Fig. 10, is 
called the "InkDivisionUnit" object 1000. This object 
1000 represents an individual element of the ink stroke 
collection resulting from the parsing operation for the 
granularity specified by an InkDivisionResult.ResultBy- 
Type operation. For example, the InkDivisionUnit object 
1000 may include an individual word of a parsed ink 
stroke collection, when the specified parsing granularity 
(or division type) was "word." 

[0104] The first property 1002 in this example object 
1000 is a Strokes property 1 004. The Strokes property 
1004 includes the strokes contained in the unit that re- 
sulted from the ink division (e.g., the strokes in the word 
or line or paragraph, depending on the selected granu- 
larity). This property 1004 gives code writers ready ac- 
cess to the strokes that make up each granular result of 
a parsing operation. The data output by or accessible 
through this Strokes property 1 004 (as indicated by out- 
put arrow 1030) may be in the following format: 

[propgetjHRESULT Strokes([out,retval]lnk- 
Strokes** Strokes) 

[0105] The property RecognizedString 1006 also is 
contained in this example of the InkDivisionUnit object 
1 000, as illustrated in Fig. 1 0. The output of this property 
1 006, which is illustrated in Fig. 1 0 by arrow 1 032, is the 
machine-generated text resulting from the handwriting 
recognition operation (and/or a pointer to a memory lo- 
cation of the data for this recognized text). 



[0106] This example of the InkDivisionUnit object 
1000 further includes a property called DivisionType 
1 008, as illustrated in Fig. 1 0. This property 1 008 returns 
the type of division unit in the object 1000 (e.g., word, 
5 line, sentence, paragraph, drawing, etc.), as illustrated 
in Fig. 10 by output arrow 1034. The data for the Divi- 
sionType property 1008 output may be in the following 
format: 

10 [propgetjHRESULT divisionType([out,retval]lnk- 
DivisionType** division Type) 

[0107] In some examples and/or uses of the inven- 
tion, this DivisionType property 1008 may be useful in 
15 the event that a given InkDivisionUnits object collection 
900 contains InkDivisionUnit objects 1000 of different 
InkDivisionTypes. 

[0108] Another property present in at least some ex- 
amples of the InkDivisionUnit object 1000 is the Rota- 

20 tionTransform property 1010. This property 1010 re- 
turns the transformation matrix information required, for 
example, to rotate the strokes in the InkDivisionUnit ob- 
ject 1 000 to horizontal. It may be used, for example, for 
rotating ink strokes in this object 1000 to a horizontal 

25 base line before sending them to a recognizer. The out- 
put data from this property 1 01 0 may take on the follow- 
ing format, at least in some examples of the invention: 

[propget] HRESULT RotationTransform([out, 
30 retvaljlnkTransform* RotationTransform 

The availability of output data from the RotationTrans- 
form property 1 01 0 is illustrated in Fig. 1 0 by output ar- 
row 1036. 

35 [0109] The RotationTransform property 1010 availa- 
ble, in at least some examples of the invention, may ex- 
pose an advantageous feature present in some appli- 
cation programs wherein handwriting that is not written 
horizontally can still be accurately parsed. In general, 
40 many known handwriting recognizers do not internally 
handle non-horizontal written lines very well, and the 
best recognition results generally are obtained in such 
products only when the baseline for the handwriting is 
horizontal. In some parser systems, however, the parser 
45 will automatically calculate or determine a baseline and 
apply a rotation transform as needed in order to obtain 
data corresponding to a horizontal line, and thereby im- 
proving the handwriting recognition capabilities of the 
handwriting recognizer. The RotationTransform proper- 
50 ty 1010 allows client code to determine whether hand- 
written information was collected horizontal. Such client 
code may use this property to "clean up" an end-user's 
handwriting by leveling it and/or to accurately draw lines 
or other shapes around the user's handwriting. 
55 [0110] In some examples of the invention, it may be 
possible for individual InkDivisionUnit objects 1000 to 
exist in the same InkDivisionUnits collection 900 but 
have different RotationTransform properties 1010. For 
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example, if a user wrote in a circle and asked for all the 
words, the InkDivisionUnits collection 900 would contain 
all the words, and each InkDivisionUnit object 1 000 may 
have a different baseline angle in its RotationTransform 
property 1010. 

[0111] Notably, the RotationTransform property 1010 
may not be relevant for all InkDivisionUnit objects 1 000. 
For example, this property 1 01 0 may be relevant for the 
InkDivisionUnit "Word" and the InkDivisionUnit "Line," 
but not necessarily for "Drawing" or "Paragraph." If de- 
sired, for the InkDivisionUnit DivisionType of "Para- 
graph," the parser could, without departing from the in- 
vention, compute a rotational angle of the paragraph 
based on its individual, constituent lines. Also, if desired, 
the parser could compute the rotational angle of a draw- 
ing, at least in some instances, from patterns observed 
in its strokes. Therefore, while the RotationTransform 
property 1 01 0 may be present for all InkDivisionTypes, 
this is not necessarily the case. 

[0112] The InkDivisionUnit object 1000 further in- 
cludes an Enum 1020 called InkDivisionType 1022, as 
illustrated in Fig. 10. The desired InkDivisionType may 
be set by client or application program code, by default, 
or in any other appropriate manner. In its output (repre- 
sented by output arrow 1038), this Enum 1022 de- 
scribes the type of InkDivisionUnits desired from a pars- 
ing operation. This information may be used, for exam- 
ple, by an InkDivision Result. ResultByType operation. 
As one example, the InkDivisionTypes for this Enum 
1020 may be defined as follows: 

InkDivisionType for Word = 0 

InkDivisionType for Line = 1 

InkDivisionType for Paragraph = 2 

InkDivisionType for Drawing = 3. 

Of course, other emums could be used without depart- 
ing from the invention. For example, an enum "Seg- 
ment" could be used to correspond to either a word or 
character, particularly for use in systems designed for 
Far East languages. 

[0113] As another example, if it is possible for a given 
InkDivisionUnits object collection to contain InkDivi- 
sionUnit objects of different InkDivisionTypes, then the 
specific values for entries in this Enum may change, for 
example, to something like the following: 

All InkDivisionTypes = 0 

InkDivisionType for Word = 1 

InkDivisionType for Line = 2 

InkDivisionType for Paragraph = 4 



InkDivisionType for Drawing = 8. 

In this manner, the InkDivisionType may become a bit- 
field and individual types can be OR'ed together to spec- 

5 ify a combination of InkDivisionTypes that are desired 
from the Result-ByType operation. 
[0114] Of course, without departing from the inven- 
tion, an InkDivisionUnit object 1000 may include prop- 
erties, methods, enums, and/or other elements in addi- 

10 tion to and/or in place of and/or in combination with the 
specific properties and enums illustrated in Fig. 10. 

V. OPERATION OF AN INK DIVIDER OBJECT AND 
API 

15 

A. Performance During an Inking Session 

[0115] In use, ink strokes may be added to, removed 
from, and/or otherwise modified within the InkDivider 

20 object 700 Strokes property 704 collection in any suita- 
ble mannerwithout departing from this invention. For ex- 
ample, if desired, a completely new Strokes property 
704 could be written and inserted each time a stroke is 
added, removed, and/or otherwise modified in an exist- 

25 ing Strokes property 704. Proceeding in this manner, 
however, would likely result in unacceptable processing 
delays as the computer would need to re-parse all of the 
ink strokes in the Strokes property 704 from scratch 
each time a stroke was changed. 

30 [0116] Accordingly, in some examples of the inven- 
tion, the client code (or application program code) that 
uses the InkDivider object 700 includes methods for: (a) 
adding individual ink strokes, (b) adding sets of ink 
strokes {e.g., by a paste operation), (c) removing ink 

35 strokes, and (d) removing sets of ink strokes (e.g., by a 
cut operation) to/from the InkDivider object's Strokes 
property collection 704, rather than replacing the entire 
Strokes property collection 704 each time a stroke is 
added, removed, or otherwise modified. By using meth- 

40 ods that add, remove, and otherwise modify only affect- 
ed strokes in a current InkDivider object's Strokes prop- 
erty 704, the internal parse tree data structure may be 
updated incrementally. For example, as a stroke is add- 
ed to a previously-parsed line (perhaps to cross a "t"), 

45 jf the new stroke is passed to InkDivider Object 700 us- 
ing an "add" method (e.g., via an "Ink Divider.Strokes. 
Add(newstroke)" operation), then the internal parser's 
parse tree data structure may be invalidated within a 
predetermined distance from the location of the new 

50 stroke(s), and thereby allow strokes only in the immedi- 
ate area of the new strokes (also called the "dirty" 
strokes or "dirty nodes") to be re-parsed, i.e., the new 
stroke associated with the letter "t" and other closely sur- 
rounding strokes, in this example. In some examples of 

55 this incremental stroke parsing, not only the actual new- 
ly added ink stroke gets parsed, but additional ink 
strokes in the neighborhood of the new stroke also get 
re-parsed taking into consideration the presence of the 
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new stroke. In some examples of the invention, a circu- 
lar region around the newly added, deleted, and/or oth- 
erwise modified stroke(s) will be reparsed (to assure 
that the newly added, deleted, or modified strokes are 
viewed in context with their surrounding strokes and to 
assure that the surrounding strokes are viewed in con- 
text with the recent modification). 
[0117] Also, in some examples, systems and meth- 
ods according to the invention perform as much parsing 
as possible in the background, without waiting for the 
Divide method to be explicitly called by the client code 
or application program code. For example, when either 
the InkDivider object's Strokes property 704 is set, or 
strokes are added to/removed from/otherwise modified 
in the InkDivider object's Strokes property 704, parsing 
occurs immediately, in a background thread on a "snap- 
shot" of the parse tree data structure, as generally dis- 
cussed above in conjunction with Fig. 4. Therefore, 
when the client code or application program code actu- 
ally calls the Divide method (e.g., at the end of an inking 
session), the InkDivider object 700 only has to (a) finish 
parsing the most recently-added/deleted/modified 
strokes (i.e., focusing on the "dirty nodes," if there are 
any), and then (b) create and return the new InkDivision- 
Result object 800. 

B. Performance When Returning to an Inking 
Session 

[01 1 8] If user applications expect to enter and exit ink- 
ing sessions multiple times and "commit" ink objects to 
the application, the InkDivider object 700 could be called 
upon to re-parse all of the existing data from scratch 
every time an application program opens an ink-contain- 
ing document. While this may be acceptable in some 
situations, performance may suffer and result in 
processing delays, particularly when opening docu- 
ments that contain large amounts of ink data. 
[0119] In at least some examples of the invention, in 
order to improve performance in these situations where 
existing documents are re-opened, the following options 
may be useful. 

1. Retaining Shadow Objects. 

[0120] As an application program exits an inking ses- 
sion and commits ink data to the screen, in at least some 
examples of the invention, it may be useful to create and 
maintain copies of the strokes in the Strokes property 
collections 1004 of each InkDivisionUnit object 1000, 
rather than cutting these original strokes from the col- 
lection. 

[0121] Applications of this nature may be designed to 
maintain shadow ink objects of all ink strokes requiring 
parsing, as well as the InkDivider object 700 and its 
Strokes property collection 704 (strokes attached to the 
ink divider) at all times. When the Divide method is 
called (e.g., at the end of an inking session), the appli- 



cation should copy, rather than cutting, the ink objects 
from the shadow object into their native application ob- 
jects. 

[0122] If any ink in the application objects is edited 
5 while not in an inking session (e.g., by performing other 
operations such as scaling or rotating), the application 
in this example of the invention may be required to re- 
move and re-add the strokes corresponding to this ap- 
plication object to the shadow collection. For example, 
10 if a drawing object provided in a word processing pro- 
gram is repositioned, the shadow ink object will need to 
be updated to reflect this repositioning. 
[0123] Upon returning to an inking session, only new 
strokes would need to be added or removed before the 
15 Divide method 710 is called again. This permits incre- 
mental processing of ink between inking sessions. 
[0124] In general, to support incremental parsing, on- 
ly a single shadow Ink object is required, such that two 
physical copies of a given Ink object exist and must be 
20 synchronized. In the case of a drawing application pro- 
gram that takes the results of the InkDivider object 700 
and creates separate Ink objects for each of the InkDi- 
visionUnit objects 1 000 (so that they can be individually 
activated and edited as drawing elements), a single 
25 shadow Ink object is required for each Ink object that 
the drawing application program creates, but overall 
there are still only two copies of the ink. 
[0125] The advantage to this method is the perform- 
ance gain when moving between inking sessions. The 
30 disadvantages are the memory footprint and burden of 
keeping collections of ink objects in sync. 

2. Reduction Heuristics 

35 [0126] Another possibility for improving parsing per- 
formance when returning to an inking session (after ex- 
iting the session) is to reduce the data set on which the 
parser is required to work. For example, it is possible to 
implement heuristics that determine whether a "commit- 
40 ted" ink object will benefit from re-parsing. As one ex- 
ample illustration, an application could limit re-parsing 
to only "committed" ink objects that intersect with new 
ink or are located with a certain spatial distance of the 
new ink. In some cases, depending on the scenario and/ 
45 or implementation, z-order of "committed" ink objects al- 
so may be a factor in re-parsing. 
[0127] While this approach initially may seem easier 
than the shadow object approach discussed above, 
care must be taken to ensure that most, if not all, of the 
50 strokes pertaining to the InkDivisionUnits that the InkDi- 
vider would modify be included in the reduction heuris- 
tics. Failure to do this could result in inconsistencies in 
the parsing results, and hence, the end-user's inking ex- 
perience. 

55 [0128] Furthermore, the internal parser itself is really 
in a better position to decide which strokes should be 
included, because it knows the invalidation scheme that 
will be used to decide what strokes to re-parse. One in- 
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them. 

(d) Reduction Heuristics: The parser's API also may 
be expanded, at least in some examples of the in- 

5 vention, to include "recommended" data set reduc- 

tion mechanisms, such that if a developer's appli- 
cation requirements were compatible, the develop- 
er would not need to worry about reduction heuris- 
tics and instead tell the parser to do its best. 

10 

(e) Parser Replacement: To allow third parties to 
use any recognizer or parser that they wish, the 
parser API may include "plug-in" support for new 
recognizers and parsers to be downloaded by the 

15 user. These alternative parsing engines would nat- 
urally have object models of their own, analogous 
to the RecognizerContext object of the API de- 
scribed above. 

20 VII. AN ALTERNATIVE INK DIVIDER OBJECT 



validation schemes uses radial invalidation, but other 
schemes may be used without departing from the inven- 
tion. 

VI. APPLICATION PROGRAMMING INTERFACES 

[0129] Numerous application programming interfaces 
("APIs") are possible to leverage various capabilities of 
the Ink API. Some examples include: 

(a) Events: In some examples of the invention, com- 
pletion of various portions of a parsing operation (e. 
g., the Divide method 800 described above) may 
trigger various events, for example, to notify the cli- 
ent code that at least a portion of the parsing oper- 
ation is completed, that at least some parsing re- 
sults are available, and/or that the parser can be 
called again. Examples of such events include 
"ParsingComplete" and "DivisionComplete" events. 
A "ParsingComplete" event, as used in some exam- 
ples of the invention, informs the application pro- 
gram that the input ink strokes have been parti- 
tioned into their respective locations in the parse 
tree data structure (e.g., like that shown in Fig. 6B). 
At this time in this example of the invention, the rec- 
ognizer has not yet acted on the ink stroke data, so 
individual word-by-word results are not yet availa- 
ble. A "DivisionComplete" event, on the other hand, 
may be used to inform the application program that 
the entire parsing and recognition operations are 
complete. In the above-described example, when a 
DivisionComplete event is fired, this tells the appli- 
cation code that an InkDivision Result object 800 is 
immediately available, and its ResultByType meth- 
od can be used to retrieve InkDivisionUnit objects 
for the desired parsing granularity. 

(b) Factoid: Factoids can be used to steer recogni- 
tion, e.g., by informing the parser of expected infor- 
mation or patterns in the ink to be recognized. For 
example, a factoid may inform a parser that an in- 
coming string was in a field for a zip code. The pars- 
er could then look for a familiar five number or nine 
number patterns. Additionally, the parser could 
preferentially recognize characters as numbers 
when coming in the zip code field (e.g., preferen- 
tially recognizing an s-shaped stroke as the number 
"5" rather than the letter "S," and/or preferentially 
recognizing the number "1 " rather than the small let- 
ter "1" or a capital "I." 

(c) Shadow Objects: The InkDivider object 700, in 
at least some examples, may provide a method to 
cause it to create an internal Shadow Ink Object to 
which it refers instead of an externally managed Ink 
object. Of course, InkDivider (or the related shadow 
ink manager object) would also provide accessors 
to the shadow Ink object(s) and methods to manage 



[01 30] Figs. 7-1 0 illustrate an example of an ink divid- 
er object and certain objects that relate to it in accord- 
ance with some examples of this invention. Fig. 11 pro- 

25 vides another example of an InkDivider object 1100. 
This sample object 1 1 00 may include various properties 
1102, methods 1104, Enums 1106, and Events 1108, 
each of which is described in more detail below. 
[0131] The "Ink" property 1110 returns/sets a refer- 

30 ence to an ink object or the strokes to be processed. 
This property 1110 is akin to the Strokes property 704 
discussed above. The "Inks" property 1112 returns acol- 
lection of ink objects generated by the Divide method 
1120 (discussed in more detail below). The Division- 
's Granularity property 1114 gets/sets the granularity by 
which the ink in the Ink property 1110 will be parsed by 
the Divide method 1120. While any default value (or 
even no default value) could be used without departing 
from the invention, in some examples of this invention 

40 the DivisionGranularity property 1114 will default to a 
"Word" granularity. The DivisionGranularity property 
1114 can be set using the DivisionGranularity Enum 
1106, which may include, for example, Enums repre- 
senting "Paragraphs," "Lines," "Sentences," "Words," 

45 "Drawings," etc. The desired DivisionGranularity may 
be set by the client or application program code, through 
default, or in any other appropriate manner. 
[01 32] The Divide method 1 1 20 performs the ink pars- 
ing operation on the ink strokes present in the Ink prop- 

50 erty 1110 based on the set DivisionGranularity property 
1114. The DivisionComplete event 1130 will be fired to 
inform the application program when the Divide method 
1120 has completed its operation. The parsed results 
from the Divide method 1120 are written to the Inksprop- 

55 erty 1112 and may be made available to the application 
program or client code from there, as illustrated by arrow 
1 1 32. Once the client code or application program code 
receives the DivisionComplete event 1130, it knows it 
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can retrieve the Inks property 1112 and retrieve the re- 
sults of the parsing operation. 

[0133] Depending on the specifics of the implemen- 
tation, the Inks property 1112 may or may not be acces- 
sible while the parsing operation (i.e., the Divide method 
1 1 20) is being performed and the internal parse tree rep- 
resentation of the ink is being constructed. 
[0134] Of course, without departing from the inven- 
tion, an InkDivider Object 1100 may include properties, 
methods, enums, events, and/or other elements in com- 
bination with and/or in addition to and/or in place of the 
specific properties, methods, emums, and events illus- 
trated in Fig. 11 . 



groupings of strokes available to the applica- 
tion program. 

2. A method according to claim 1, wherein the infor- 
5 mation made available to the application program 

includes at least one of the one or more groupings 
of strokes. 

3. A method according to claim 1, wherein the infor- 
10 mation made available to the application program 

includes information indicating a number of group- 
ings of strokes having the first predetermined gran- 
ularity. 



VIII. CONCLUSION 

[0135] While the invention has been described in 
terms of various specific examples, these specific ex- 
amples merely exemplify the invention and do not limit 
it. Those skilled in the art will recognize, for example, 
that while various specific names are used for objects, 
properties, methods, enums, events, and the like in this 
specification, these specific names are merely exam- 
ples of possible names and should not be construed as 
limiting the invention. Of course, other names may be 
used for objects, properties, methods, emums, events, 
and the like without departing from this invention. Addi- 
tionally, the specific arrangement of objects, properties, 
methods, enums, events, and like may differ from the 
specific arrangements described and illustrated without 
departing from the invention. 

[0136] Additionally, the fact that a specific feature or 
function of the invention is described in conjunction with 
a specific example does not mean that this feature or 
function is limited to use with that specific example of 
the invention or that every example must include that 
specific feature or function. Rather, unless otherwise 
specified; the various features and functions described 
above may be used freely in any example of the inven- 
tion. Those skilled in the art will appreciate that changes 
and modifications may be made to the exemplified ver- 
sions of the invention without departing from the spirit 
and scope of the invention, as defined in the appended 
claims. 



Claims 

1. A method of making information available to an ap- 
plication program, comprising: 

storing a plurality of ink strokes; 

issuing a divide request; 

in response to the divide request, grouping the 

stored ink strokes into one or more groupings 

of strokes having at least a first predetermined 

granularity; and 

making information regarding the one or more 



15 4. a method according to claim 1, wherein the infor- 
mation made available to the application program 
includes machine-generated text that corresponds 
to at least one of the one or more groupings of 
strokes. 

20 

5. A method according to claim 1 , further comprising: 

informing the application program when the 
grouping has been completed. 

25 

6. A method according to claim 1, wherein at least a 
portion of the information regarding the one or more 
groupings of strokes made available to the applica- 
tion program is stored in an inkdivision result object. 

30 

7. A method according to claim 1, wherein language 
is at least one factor considered in grouping the 
stored ink strokes into one or more groupings of 
strokes having the first predetermined granularity. 

35 

8. A method according to claim 1 , wherein the first pre- 
determined granularity is selected from the group 
consisting of: word, line, paragraph, sentence, and 
drawing. 

40 

9. A method according to claim 1 , wherein the group- 
ing additionally groups the stored ink strokes into 
one or more groupings of strokes having at least a 
second predetermined granularity. 

45 

10. A method according to claim 1 , further comprising: 

making information regarding one or more 
groupings of strokes having a second predeter- 
50 mined granularity available to the application 

program. 

1 1 . A method according to claim 1 , further comprising: 

55 changing the stored ink strokes to produce a 

modified ink stroke set; 
issuing a second divide request; and 
in response to the second divide request, 
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grouping the modified ink stroke set into one or 
more groupings. 

12. A computer-readable medium having computer-ex- 
ecutable instructions for performing a method of 
making information available to an application pro- 
gram, the method comprising: 

storing a plurality of ink strokes; 

receiving a divide request; 

in response to the divide request, grouping the 

stored ink strokes into one or more groupings 

of strokes having at least a first predetermined 

granularity; and 

making information regarding the one or more 
groupings of strokes available to the applica- 
tion program. 

13. A computer-readable medium according to claim 
12, wherein the information made available to the 
application program includes at least one of the one 
or more groupings of strokes. 



20. A computer-readable medium according to claim 
12, wherein the grouping additionally groups the 
stored ink strokes into one or more groupings of 
strokes having at least a second predetermined 

5 granularity. 

21. A computer-readable medium according to claim 
12, wherein the method further includes: 

10 making information regarding one or more 

groupings of strokes having a second predeter- 
mined granularity available to the application 
program. 

15 22. A computer-readable medium according to claim 
12, wherein the method further includes: 

receiving a modified ink stroke set; 
receiving a second divide request; and 
20 in response to the second divide request, 

grouping the modified ink stroke set into one or 
more groupings. 



14. A computer-readable medium according to claim 

12, wherein the information made available to the 25 
application program includes information indicating 
a number of groupings of strokes having the first 
predetermined granularity. 

15. A computer-readable medium according to claim 30 
12, wherein the information made available to the 
application program includes machine-generated 
text that corresponds to at least one of the one or 
more groupings of strokes. 

35 

16. A computer-readable medium according to claim 
12, wherein the method further includes: 



23. A method of communicating between an application 
and an ink divider object, the ink divider object stor- 
ing ink strokes to be divided into groups, compris- 
ing: 

issuing adivide request to the inkdivider object; 
in response to the divide request, calling a di- 
vide method, which groups the stored ink 
strokes into one or more groupings of strokes 
having at least a first predetermined granulari- 
ty; and 

making information regarding the one or more 
groupings of strokes available to the applica- 
tion. 



informing the application when the grouping 
has been completed. 

17. A computer-readable medium according to claim 
12, wherein at least a portion of the information re- 
garding the one or more groupings of strokes made 
available to the application program is stored in an 
ink division result object. 

18. A computer-readable medium according to claim 
12, wherein language is at least one factor consid- 
ered in grouping the stored ink strokes into one or 
more groupings of strokes having the first predeter- 
mined granularity. 

19. A computer-readable medium according to claim 
12, wherein the first predetermined granularity is 
selected from the group consisting of: word, line, 
paragraph, sentence, and drawing. 



24. A method according to claim 23, wherein the infor- 
40 mation made available to the application includes 

at least one of the one or more groupings of strokes. 

25. A method according to claim 23, wherein the infor- 
mation made available to the application includes 

45 information indicating a number of groupings of 
strokes having the first predetermined granularity. 

26. A method according to claim 23, wherein the infor- 
mation made available to the application includes 

50 machine-generated text that corresponds to at least 
one of the one or more groupings of strokes. 

27. A method according to claim 23, further comprising: 

55 informing the application when the divide meth- 

od has been completed. 

28. A method according to claim 23, wherein at least a 
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portion of the information regarding the one or more 
groupings of strokes made available to the applica- 
tion is stored in an ink division result object. 

29. A method according to claim 23, wherein during op- 5 
eration of the divide method, language is at least 
one factor considered in grouping the stored ink 
strokes into one or more groupings of strokes hav- 
ing the first predetermined granularity. 

10 

30. A method according to claim 23, wherein the first 
predetermined granularity is selected from the 
group consisting of: word, line, paragraph, sen- 
tence, and drawing. 

15 

31 . A method according to claim 23, wherein during op- 
eration of the divide method, the divide method ad- 
ditionally groups the stored ink strokes into one or 
more groupings of strokes having at least a second 
predetermined granularity. 20 

32. A method according to claim 23, further comprising: 

making information regarding one or more 
groupings of strokes having a second predeter- 25 
mined granularity available to the application. 

33. A method according to claim 23, further comprising: 

changing the stored ink strokes to produce a 30 
modified ink stroke set; 

issuing a second divide request to the ink divid- 
er object; and 

in response to the second divide request, call- 
ing the divide method, which groups the modi- 35 
fied ink stroke set into one or more groupings. 

34. A computer-readable medium having computer-ex- 
ecutable instructions for performing method of com- 
municating between an application and an ink divid- 40 
er object, the ink divider object storing ink strokes 

to be divided into groups, the method comprising: 

issuing adivide requesttothe inkdividerobject; 
in response to the divide request, calling a di- 45 
vide method, which groups the stored ink 
strokes into one or more groupings of strokes 
having at least a first predetermined granulari- 
ty; and 

making information regarding the one or more 50 
groupings of strokes available to the applica- 
tion. 

35. A computer-readable medium according to claim 

34, wherein the information made available to the 55 
application includes at least one of the one or more 
groupings of strokes. 



36. A computer-readable medium according to claim 
34, wherein the information made available to the 
application includes information indicating a 
number of groupings of strokes having the first pre- 
determined granularity. 

37. A computer-readable medium according to claim 
34, wherein the information made available to the 
application includes machine-generated text that 
corresponds to at least one of the one or more 
groupings of strokes. 

38. A computer-readable medium according to claim 
34, wherein the method of communicating further 
includes: 

informing the application when the divide meth- 
od has been completed. 

39. A computer-readable medium according to claim 
34, wherein at least a portion of the information re- 
garding the one or more groupings of strokes made 
available to the application is stored in an ink divi- 
sion result object. 

40. A computer-readable medium according to claim 
34, wherein during operation of the divide method, 
language is at least one factor considered in group- 
ing the stored ink strokes into one or more group- 
ings of strokes having the first predetermined gran- 
ularity. 

41. A computer-readable medium according to claim 
34, wherein the first predetermined granularity is 
selected from the group consisting of: word, line, 
paragraph, sentence, and drawing. 

42. A computer-readable medium according to claim 
34, wherein during operation of the divide method, 
the divide method additionally groups the stored ink 
strokes into one or more groupings of strokes hav- 
ing at least a second predetermined granularity. 

43. A computer-readable medium according to claim 
34, wherein the method of communicating further 
includes: 

making information regarding one or more 
groupings of strokes having a second predeter- 
mined granularity available to the application. 

44. A computer-readable medium according to claim 
34, wherein the method of communicating further 
includes: 

receiving changes to the stored ink strokes to 

produce a modified ink stroke set; 

issuing asecond divide request to the ink divid- 
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er object; and 

in response to the second divide request, call- 
ing the divide method, which groups the modi- 
fied ink stroke set into one or more groupings. 

5 

45. A computer-readable medium having a data struc- 
ture stored thereon for an ink divider object com- 
prising: 

a first data set defining a strokes property for 10 
storing information regarding one or more ink 
strokes; and 

a second data set defining a divide method for 
grouping the one or more ink strokes into one 
or more groupings of strokes having at least a 15 
first predetermined granularity. 

46. A computer-readable medium according to claim 
45, wherein the ink divider object further includes a 
third data set defining a recognizer context property 20 
for storing information regarding a recognizer to be 
used in dividing the one or more ink strokes. 

47. A computer-readable medium having a data struc- 
ture stored thereon for an ink division result object 25 
comprising: 

a first data set defining a strokes property for 
storing information regarding one or more ink 
strokes that have been subjected to a divide 30 
method and grouped into one or more group- 
ings having at least a first predetermined gran- 
ularity; and 

a second data set defining a result by type 
method for enabling retrieval of groupings of ink 35 
strokes of at least the first predetermined gran- 
ularity. 

48. A computer-readable medium according to claim 

47, wherein the result by type method enables re- 40 
trieval of groupings of ink strokes at a plurality of 
different granularities including the first predeter- 
mined granularity. 

49. A computer-readable medium having a data struc- 45 
ture stored thereon for an ink division unit object 
comprising: 

a first data set defining a strokes property for 
storing information regarding one or more ink 50 
strokes contained in a grouping of one or more 
ink strokes; and 

a second data set defining a division type for 
storing information regarding a type of ink 
strokes contained in the grouping. 55 

50. A computer-readable medium according to claim 
49, wherein the data structure further includes a 



third data set defining or pointing to machine-gen- 
erated text corresponding to the one or more ink 
strokes contained in the strokes property. 

51. A computer-readable medium according to claim 

50, wherein the data structure further includes a 
fourth data set defining an angle of rotation neces- 
sary to rotate the one or more strokes contained in 
the strokes property to an angle suitable for intro- 
duction into a handwriting recognition system. 

52. A computer-readable medium according to claim 

51 , wherein the datastructure further includes afifth 
data set defining an enumerated value correspond- 
ing to the division type. 

53. A computer-readable medium according to claim 
49, wherein the data structure further includes a 
third data set defining an angle of rotation neces- 
sary to rotate the one or more strokes contained in 
the strokes property to an angle suitable for intro- 
duction into a handwriting recognition system. 

54. A computer-readable medium according to claim 
49, wherein the data structure further includes a 
third data set defining an enumerated value corre- 
sponding to the division type. 

55. A computer-readable medium having a data struc- 
ture stored thereon for an ink division units object 
comprising: 

a first data set defining a count of stroke group- 
ings present in an ink object; and 
a second data set defining an item method that 
retrieves a specific stroke grouping given an in- 
dex value for the specific stroke grouping. 

56. A method of communicating between a parser and 
an application program, comprising: 

sending data representing a plurality of ink 
strokes from the application program to the 
parser; 

providing parsing information to the parser; 
grouping the ink strokes into one or more 
groupings of strokes having at least a first pre- 
determined granularity; and 
making information regarding the one or more 
groupings of strokes available to the applica- 
tion program. 

57. A method according to claim 56, wherein providing 
the parsing information includes setting a recogniz- 
er to be used in the grouping. 

58. A method according to claim 56, wherein providing 
the parsing information includes setting a language 
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to be used in the grouping. 

59. A method according to claim 56, wherein providing 
the parsing information includes setting the first pre- 
determined granularity. 5 

60. A method according to claim 56, wherein providing 
the parsing information includes setting an expect- 
ed line height for lines of text included in the ink 
strokes. 10 

61 . A computer-readable medium that includes compu- 
ter-executable instructions stored thereon for per- 
forming the method of claim 56. 
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